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Research across five countries has identified inability to pay attention to mathematical detail 
— the discipline of noticing — is an issue in senior secondary school mathematics teachers. 
The test and questionnaire completed by an Australian cohort further identifies a reluctance 
to employ non-routine questions in assessments, with teachers concerned about damaging 
the trust relationship they enjoy with their students. As teachers fail to demonstrate strong 
ability in non-routine written test questions themselves, this paper questions whether there 
exists a ‘conspiracy’ between teachers and their students to avoid scrutiny of conceptual 
understanding. 


Introduction 


In mathematics education, systematic reflective thinking may involve interrogation of 
practice both inwardly and outwardly (Mason, 2002). Inward reflection may be practical, a 
consideration of such matters as classroom management and appropriate delivery of the 
curriculum. This may lead to a metacognitive reflection, as the practitioner searches for the 
causes of their confidence in the techniques they are applying and a search for their personal 
assumptions and abilities. At a higher stage again, Mason (2002, p. 17) identifies “social- 
reflection”, a more outward-directed critique of the values which impose upon the teaching 
situation. 

A key concept in reflective thinking is intentional noticing. Mason (2002) relates an 
anecdote about pianist Artur Rubinstein deliberately choosing to not use a certain finger in 
a concert, just to be more aware of his playing. In teaching practice, self-noticing can be 
practiced in respect of gesture, how conversation is initiated or terminated, or of the things 
the practitioner chooses to note down in writing. Underlying Mason’s approach is the idea 
that, through conscious practise, noticing and ultimately teaching performance can be 
improved. 

This study seeks to apply Mason’s (2002) concepts of intentional noticing to assessment 
of mathematics. Incorrect responses to mathematical questions can result from overlooking 
aspects of the problem. For example, in calculus a local maximum may be obtained by 
differentiation when, for the defined domain, the global maximum may be greater and 
optimal. Overlooking discontinuities in functions is another difficulty in the mathematical 
performance of school students. In the opaque language of Examiners Reports, a statement 
such as “Standard questions involving calculus, logarithms and the exponential function 
were well attempted” (School Curriculum and Standards Authority, 2018, p. 1) can be taken 
to indicate that the unmentioned non-standard questions were not subject to sufficiently close 
attention. 

This paper recounts research by Klymchuk (2014) on the phenomenon of lack of 
attention in mathematics students and in their teachers. It describes replication in an 
Australian setting of Klymchuk’s study and presents an account of the self-knowledge of the 
Australian teachers involved. Klymchuk’s conclusion is that “Solving non-routine, non- 
standard questions would better prepare students for the real world. Enhancing their own and 
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their students’ discipline of noticing by paying attention to details can also be a useful 
addition to teachers’ professional development.” (p. 69). This paper explores whether these 
sentiments apply in an Australian situation and reports the self-interpretation of the actors 
involved. It explores whether the intentional noticing of Mason (2002) may be a means of 
improving the mathematical performance of teachers and students in Australia. 


Theoretical Framework and Literature Review 


Intentional noticing appears under several names in educational research. Sch6n (1987) 
favoured the term “reflection”, but partitioned it into such categories as “reflection-in- 
action”. Mason (2002) offers further varieties of reflection, examining intention and 
motivation. Noticing is treated as a study of student misconceptions in works including Ryan 
and Williams (2007). More recently, mindfulness in mathematics is a theme of Boaler 
(2016), developing work by Dweck (2007). 

Duit, Treagust and Mansfield (1996) note that from a constructivist perspective there is 
a symmetrical relationship between teacher and school students, both being partners in a 
communication and trying to obtain an idea of the understanding of the other. With this in 
mind, it is important to examine the self-perceptions of the teachers, not just the performance 
of the students, and also the joint behaviour and understandings of teachers and students. 

To move from habitual or mechanical patterns in teaching practice, Mason (2002) 
recommends teachers undertake a series of exercises to develop increased sensitivity. These 
include mirroring gestures, listing key words for a lesson and introspection about 
achievement of desired intentions in interactions with students. This same intentional 
improvement in performance is possible in mathematics, claim Meyer, Falkner, 
Sooriamurthi and Michalewicz (2014). Their solution is exposure to carefully-selected 
mathematical puzzles where close attention is needed. 

Puzzle-based Learning is rapidly becoming a bigger and bigger part of the curriculum as there is no 

guarantee that a traditional education will provide students with enough practise and experience to 

develop problem-solving skills. The rapidly changing face of employment and technology means that 

the problems that we train people to solve today are probably not the problems they will be solving 

in ten years. When our current education system tends to favour highly focused learning of rigid 

approaches to predictable problem sets, there is no guarantee that our students will be flexible enough 


and resilient enough to cope with open-ended problems with no guaranteed solution. (Meyer et al., 
2014, p. 4) 


Research originally undertaken by Klymchuk (2014) in New Zealand was then was 
replicated by his associates in three other countries: Hong Kong, Germany and Ukraine. 
Klymchuk describes the New Zealand group as “experienced upper secondary school 
mathematics teachers”; the Hong Kong group as “secondary school mathematics teachers”; 
the German group as “experienced school mathematics teachers”; and the Ukraine group as 
“[Tertiary] Year 3-4 mathematics students training to become secondary school mathematics 
teachers with the majority having had teaching experience as part of their training” (p. 64). 
Klymchuk distinguished two clusters: in the New Zealand and German groups “roughly half 
of the participants were disappointed and embarrassed while the other half were more 
positive and saw the opportunity for improvements”, whereas the Hong Kong and Ukraine 
groups “the vast majority were very disappointed and uncomfortable”. Klymchuk indicated 
“The difference between the two clusters might be due to culture” (p. 67). 

Klymchuk (2014) designed a test and questionnaire to explore the role of reflective 
thinking in mathematics assessment and to search for the underlying reasons for incorrect 
answers. The test questions — as given in Appendix A, with solutions — are described by 
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Klymchuk as “provocative” in the sense that, although they appear routine, each of the seven 
questions contains a non-routine ‘catch’ which relies on conditions and constraints within 
the mathematics. For example, the third question asks respondents to “Solve the equation 
In(x? + 17x — 18) — In(x? +5x —6) =0”. The initial impression may be that this 
question will succumb to standard mathematical manipulations, however the domain of the 
logarithm function is restricted to numbers greater than zero, therefore no real value of x 
satisfies the equation. 

Mason’s (2002) “discipline of noticing” involves paying attention to such detail as 
conditions, constraints, locality, properties and relationships. It was expected that the various 
‘catches’ in Klymchuk’s test would result in some participants obtaining incorrect answers, 
even though some countries were represented by experienced mathematics teachers. In 
practice, Klymchuk (2014, p. 63) reported that “The results of the test were startling — the 
vast majority of the participants gave incorrect answers to most questions in the test.” 


Methodology 


In order to replicate the research undertaken by Klymchuk (2014) in an Australian 
setting, an account of the exact conditions under which the test and questionnaire were 
administered was obtained. The English language version of the test and questionnaire were 
adopted verbatim. The Australian survey participants were delegates at a November 2017 
conference of secondary mathematics teachers, thereby ensuring all were practicing 
secondary mathematics teachers. The participants were invited to take part in a conference 
session entitled “An experiment which may change your teaching practice” with this session 
abstract: “We will do a test which contains some routine questions and some trick ones. 
Some are calculus questions. We will discuss the answers and consider the implications for 
our teaching.” The expectation was that the mention of calculus would dissuade attendance 
by mathematics teachers whose experience was only of junior secondary classes. 

Sixty teachers attended, completing the 15-minute test and a short questionnaire about 
their test results after the solutions had been discussed. In order to maintain conformity with 
the research in other countries, the wording of the session abstract, test questions, solutions 
and questionnaire prompts were used verbatim as reported in the original study (Klymchuk, 
2014), with the same procedure and timing. The seven-question test was distributed as a 
write-on paper while a 15-minute countdown timer was visible on a projector screen. 
Although participants were seated adjacent to each other around large tables, no discussion 
was permitted and no such cooperation was observed. 

In Western Australia three of the senior school mathematics courses have summative 
examinations to which school-based assessments are moderated: Mathematics Specialist, 
Mathematics Methods and Mathematics Applications. These are collectively titled the 
ATAR [Australian Tertiary Admission Rank] courses. The test questions discussed in this 
paper are based on content from ATAR mathematics courses. As evident from their test 
responses, in four cases the teachers demonstrated no acquaintance with the content of the 
ATAR courses, only with algebra. Although they were attending a conference for secondary 
mathematics teachers, these four may teach only junior classes and / or may be teaching out- 
of-field without a tertiary mathematics background. 
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Mathematics Test Results 


The percentage of correct answers for each question, calculated against the number of 
completed test papers, is presented in Table 1. The percentages are a comparison of the 
number of correct solutions for the question, marked either fully correct or incorrect, against 
the country cohort size n. Klymchuk (2014) is the source of the non-Australian data. 


Table 1 


Percentage of Correct Solutions in Non-Routine Senior Secondary Mathematics Questions 
Question number 


Country n 1 2 3 4 5 6 7 


Australia a7 2 54 33 4 28 9 28 
Germany 10 0 60 30 NA 20 0 0 
Hong Kong 26 Zz {2 ZI 19 12 15 12 
New Zealand 14 0 7 pa 7 0 8 0 
Ukraine 8 26 0 19 al is) i 0 


In the Australian study, 60 people were present to undertake the test, but only 57 test 
papers were received. Of these, there were 17 teachers who failed to obtain any correct 
solutions. It was evident from blank responses to questions that many of the Australian 
teachers had little idea of how to approach the questions at all, and the various ‘catches’ 
proved difficult even for those who were able to make a start. As in the research in other 
countries, no information was gathered directly from participants about their extent of 
training in mathematics nor length of experience as mathematics teachers. 

Several misdirecting factors may have contributed to low scores. Despite indication in 
the session abstract that “some” trick questions would be included, in fact what Klymchuk 
(2014) refers to as ‘catches’ existed in every one of the seven questions. Some indicative 
questions may be seen in Appendix A. In Q1 the phrasing “height dropped on the 
hypotenuse” may have been unfamiliar to some participants. Q3 and Q5 demanded solving 
where there is no solution. Q4 directs participants to “prove the identity” of an equation 
which is not an identity at all. Q6 and Q7 ask participants to “find” items which actually do 
not exist. Such directly-phrased instructions may have been taken by the Australian cohort 
as indication that the task is feasible. Also, a typographical error (the symbol for “equals 
zero” was omitted) existed in Q5 until corrected verbally during the test. 


Questionnaire Results 


When the test papers were collected in, solutions were distributed and discussed and a 
follow-up questionnaire was distributed. 51 completed questionnaires were received. The 
questionnaire paper contained just three prompts: “What are your feelings after you have 
learnt about the correct solutions to the test questions?”, “What are the reasons for not 
solving all test questions correctly?” and “Would you make any changes in your teaching 
practice after doing the mini-test. If so — what changes? If not — why?”. Fifty-one completed 
questionnaires were received. They are reported here without regard to the prompt as the 
written responses often were not specific to a single prompt. The themes which emerged are 
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frustration, self-awareness, a trust relationship between teachers and students, and a 
perception that procedural understanding is prioritised by the examination system. 

The responses were phrased politely but they do give an idea of the degree of 
exasperation and concern felt by some of the teachers: “Annoyed at being tricked by some 
obvious things”, “I’ve been knocked down a peg or two. While doing the questions I felt 
confident in my ability and did spot a couple of the conundrums but it turns out that most of 
them got past me”, “We were stitched up followed by great learning opportunity”, “Ashamed 
because I did not get the correct answers’, “I think this is an ‘immoral’ test. I don’t believe 
you should ask someone to prove something is true when it isn’t”, “Make sure questions 
work and don’t be nasty”, “I didn’t pay attention to the fundamental assumptions for a 
concept!”’, “I was too busy doing the mechanics of the questions and didn’t take time to look 
carefully at all parts of the questions presented to see if they truly exist”, “I would like to re- 
sit the test so that I can check if I have learnt from errors”, “Tricked”, “A little foolish, 
surprised”, “Frustration, disappointment’, “Not paying attention”, “Excitement, satisfaction, 
curiosity”. Responses such as “Not paying attention” reflect Klymchuck’s (2014) subtitle 
“drawing attention to a lack of attention”, but other responses do not. Many Australian 
respondents did not identify in themselves a lack of attention to mathematical detail, but 
rather they identified improper questions — questions which did not accord with the 
expectations of the respondents because they incorporated difficult and unexpected 
elements. These are elements which required noticing of detail. The Australian cohort are 
best associated with the New Zealand and German cluster, where there is divergence of 
response, rather than the Hong Kong and Ukraine cluster where subjects focused strongly 
on deficiencies in their own performance. This may indicate “social-reflection” values as 
identified by Mason (2002) or “culture” as identified by Klymchuk (2014). 

In the Australian cohort, a dozen respondents (24%) in their questionnaire comments 
noted their own lack of knowledge, often coupled with indication of the reasons: “Lack of 
practice — have not studied / applied maths at this level for 12 years’, “I had forgotten some 
concepts which I haven’t used in a long time, as I don’t teach methods & specialist at the 
moment”, “I am not maths trained”, “Didn’t know the topic to that depth”, “I failed to see 
each question had underlying principles or assumptions that I did not remember”. 

The word “trust” featured in six responses (12%). It was used to indicate that teaching 
and assessment is a bond between teachers and students. “Students should be able to trust 
the questions”, “When students are doing a test they shouldn’t be looking for a trick all the 
time. It is a matter of trust.”, “[I have] basic trust test questions are mostly correct!!!", “I 
ignored whether the solutions were possible, i.e. I trusted the questions were “real” and 
[solutions] existed”, “Trust. Didn’t check all solutions. I did pick up some issues and became 
suspicious”, “I didn’t even think to check whether or not the triangle was possible as there 
was a “trust” that the triangle is real if the examiner is asking for an area. ... Q5 I showed 
that there are no solutions but crossed out my working, again having a “trust” in the 
examiner’s questions”. In a similar vein, other responses included: “In my teaching practice 
my students are given only problems which are possible to solve. They follow the script.”, 
“T would hope that the majority of my questions are already checked and are workable as 
they are.” “At school we don’t tend to pose impossible questions”, “I feel a teacher giving 
these questions would be unfair”. At the conclusion of the session, one teacher approached 
the investigator to share that such ‘trick’ questions are inappropriate for school students 
because they “destroy the trust of my students”. She encourages young teachers to work the 
examination questions themselves so that no impossible questions are presented. Another 
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teacher privately intimated that he always tells the students there are no ‘trick’ questions in 
the test, otherwise the students waste too much time looking for them. 

One factor which figured prominently in the responses was that the teachers saw 
preparation for examinations as the pre-eminent priority: “Exams always have questions 
which make sense, so why teach them beyond the process?”, “I am worried that in an 
assessment they will become absorbed by looking for the trick and waste precious time, as 
the assessments they do, do not have trick questions.”, “We are trying to get them to be 
successful in their WACE [West Australian Certificate of Education] exams after all.” 


9 66. 


Twenty-five respondents (49%) used the near-equivalent words “process”, “mechanics”, 
“methods”, “procedure”, “algorithms”, “routines” and “strategies”. These terms were used 
to indicate that the test-taking activity was viewed as a predictable pathway. Gaining 
familiarity with this pathway was preparation for examinations. But, as one teacher noted, 
“Tt is very easy to follow a rule / algorithm / formula, but unless you have the understanding 
‘why’ you cannot see when there may be no solution”. 

A need for students to “question the questions” was recommended by 27 respondents 
(53%) who indicated that they will make use of such ‘trick’ questions, but only with their 
most able students for two respondents. Conversely, 23 respondents (45%) indicated it is 
contingent on teachers to ensure all questions have feasible solutions. These respondents did 
not see value in ‘trick’ questions which call for greater scrutiny of the conceptual 
underpinnings of the content, or they felt such considerations should not be included in 
assessments. 


Discussion 


The key observation about the overall results in Table 1 is that many teachers were 
unable to answer the questions. In many of the test questions it was not the ‘catch’ which 
caused Australian respondents to fail the question, it was lack of application of the routine 
methods of solution. In Question 5, for example, 28 of the 57 teachers did not employ the 
Intermediate Value Theorem nor any other productive means of attack. It was not that the 
function contains a discontinuity which was problematic: the teachers were unaware of basic 
function analysis technique in the first place. 

Subtle linguistic cues may influence teachers in different cultures and operating in 
different languages. For some people it may be discourteous to respond “this cannot be done” 
to a question posed in a university-warranted test. Australian teachers may have little 
experience in mathematical questions which have no solution — or a multiplicity of solutions 
— whereas the format may not be unconventional in other countries. 

If senior secondary mathematics assessments contain few questions which explore full 
understanding, the questions presented to “trusting” students must be routine, the content 
pre-negotiated and expected. Students who are not exposed to the risk of losing their self- 
confidence are therefore not being challenged in assessments to demonstrate more than 
procedural competence. This suggests there may be a conspiracy between teachers and their 
students to avoid coverage of true conceptual understanding in senior school mathematics 
tests and examinations. 

The idea of ‘conspiracy’ in relation to the assessment of mathematics in Australia has 
appeared before. The architect of the current mathematics curriculum, Professor Peter 
Sullivan, writes on the use of open-ended tasks: 


One of the major constraints that teachers experience when utilising such tasks is that many students 
avoid risk taking and do not persist with the challenges that are required in order to complete the task. 
And teachers are sometimes complicit in this avoidance strategy. Desforges and Cockburn (1987), for 
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example, reported on a detailed study of primary classrooms in the United Kingdom and found that 
students and teachers conspired with each other to reduce the level of risk for the students. (Sullivan, 
2010, p. 38) 


Although Sullivan (2010) discusses primary students, his point is that teachers can be 
complicit in an avoidance strategy. 

The local mathematics curriculum (School Curriculum and Standards Authority [SCSA], 
n. d.,) states in the Rationale of each of the ATAR curricula that “For all content areas of the 
... course, the proficiency strands of the Year 7—10 curriculum continue to be applicable and 
should be inherent in students’ learning of the course. These strands are Understanding, 
Fluency, Problem-solving and Reasoning ...”. The curriculum requirements include that 
students “interpret mathematical information and ascertain the reasonableness of their 
solutions to problems”, a phrase repeated in the Learning Outcomes section in every Unit of 
the ATAR curricula. Senior secondary students therefore should encounter content such as 
boundary conditions, discontinuities and limitations to definitions. Provision exists within 
senior secondary mathematics school-based assessment to provide investigation of such 
topics as the limitations on and counter-intuitive properties of functions. For all three ATAR 
courses the curriculum (SCSA, n. d.) notes of the school-based Investigation tasks “This 
assessment type provides for the assessment of general inquiry skills, course-related 
knowledge and skills, and modelling skills.” If teachers have only a procedural 
understanding of mathematics themselves, the school-based assessments they provide may 
just mimic examinations and fail to include these wider skills. 

The question arises whether the ATAR examinations themselves could do more to allow 
“paying attention” as Klymchuk (2014) phrases it. In the 2017 Mathematics Methods 
examination (School Curriculum and Standards Authority, 2017) Q9 requires calculation of 
a linear model but later allows students to reject the model on the basis of patterns in 
residuals; Q10 asks students to select appropriate “equation(s)” where four are offered and 
two of them are appropriate; Q12 has marks for students declaring a given interpretation is 
incorrect because “cause is not established”; Q14 “find any point of inflection” has no point 
of inflection because the function is undefined at the only value of x where the second 
derived function is equal to zero. Unlike Klymchuk’s questions, the examination does not 
direct students to find solutions which do not exist, and there is little of the “noticing” 
advocated by Mason (2002). An in-depth study of Australia’s senior secondary mathematics 
teaching and examinations is beyond the scope of this paper, but the examination cited 
provides very mild support, if any, that the questions allow students to employ Mason’s 
“noticing”. 


Conclusions 


The Australian teachers had difficulty recognising ‘trick’ questions, and many expressed 
a disinclination to employ such questions in their own teaching. This study reveals that a 
significant number of teachers are unable to tackle hard mathematics questions on a routine 
basis in the first place. Avoidance of difficult material is not in the best interests of students 
and defensive responses by teachers may indicate a need for increased provision of in-service 
professional learning. Senior secondary teacher preparation courses mandate tertiary-level 
mathematics units. But tertiary mathematics does not constitute the material taught in 
secondary schools. Passing a tertiary mathematics unit does not ensure deep understanding 
of concepts in secondary school mathematics, an area in which secondary mathematics 
teachers may be wanting. 


192 


Australian teachers greatly value the trust relationship they enjoy with their students. 
However, this seems to deter some teachers from utilising questions which call on students 
to demonstrate deep conceptual understanding and confident exhibition of self-belief. 
Greater use of puzzle-based learning and intentional noticing may well prove advantageous 
in Australian schools. 
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Appendix A 


Indicative “provocative” questions posed by Klymchuk (personal communication, March 13, 2017). 


2. Find the domain of the function y = f(g(x)) if f(x) =x? +1, g(x) =vx—2 
3. Solve the equation In(x? + 17x — 18) — In(x? + 5x —6) =0 

5. Show that the equation mae =0 has a solution on the interval [0, 2] 

6. Find the derivative of the function y = In(2sin(3x) — 4) 
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